% N79-15612 


3 > 




HUMAN /COMPUTER CONTROL OP UNDERSEA TELEOPERATORS 

T.B. Sheridan, W.L. Verplank and T.L. Brooks 

Kan-Machine Systems Laboratory 
Department of Mechanical Engineering 
Massachusetts Institute of Technology 
Cambridge. Massachusetts 02139 


Abstract 

This paper discusses the potential of supervisory controlled tele- 
operators for aceeeplishmsnt of manipulation and sensory tasks in deep 
ocean environments, and discusses one such system. Teleoperators and 
supervisory control are defined, the current problems of hwan divers are 
reviewed, and some assertions are made about why supervisory control has 
potential use to replace and extend human diver capabilities. The relative 
roles of man and computer and the variables involved in man-computer inter- 
action are next discussed. Finally, a detailed description of a supervisory 
controlled teleoperator system, SUPERMAN, is presented. 


1. Teleoperators and Supervisory Control 

Many future undersea tasks may be aceespllshed by "teleoperators". 

We define teleoperators to be general purpose submersible work vehicles 
controlled remotely by human operators and tilth video and/or ether sensors, 
power and propulsive actuators for mobility, with mechanical hands and arms 
for manipulation and possibly a computer for a limited degree of control 
autonomy. A manned s u bm e rsible is not a teleoperator vehicle, but its 
attached manipulators are certainly teleoperators, requiring control through 
a viewing port or through closed-circuit video. Sometimes the term "tele- 
operator" is restricted to telemanipulator, excluding the system for re- 
motely positioning and orienting a sensor, but for the sake of generality 
we include this important function. 

This paper focuses on those aspects of undersea teleoperation which 
concern the human operator and the man-machine interface, and within this 
still relatively broad domain, it concentrates on the prospects for utili- 
sation of "supervisory control". Supervisory control is a hierarchical 
control scheme whereby a system (which could be a teleoperator, but could 
also be an aircraft, power plant, etc.) having sensors, actuators and a 
computer, and capable of autonomous decision-taking and control over short 
periods and in restricted conditions, is remotely monitored and inter- 
mittently operated directly or r e pr o gram m ed by a person. 

The distinction between direct humurt control of a teleoperator and 
supervisory control of a teleoperator is made graphically in Figure l. In 
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Figure 1. 


Definition* 

TELEOPERATOR 


Definition* of Teleoperetor end Supervisory Control 
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A vehicle having aeneore and aetuatora for nobility and/or manipulation 
controlled by a human operator, and thus enabling him to extend himself to 
physically remote or hasardous environment*. 


SUPERVISORY CONTROL 

A helrarehlcal control scheme vherehy a device having sensors, actuators 
sod a computer, and capable of autonomous decision ashing and control over short 
periods and restricted conditions is remotely monitored and Intermittently operated 
directly or reprogrammed by a person. 




the upper figure the humn directly control*, over eitlier a wire or sonic 
communication link, the separate propulsive actuators of the vehicle, the 
actuators for the separate degrees of freedom of the manipulator, and the 
pan and tilt actuators of the video camera. The video picture is sent 
back directly to the operator. The ••hand control" can be a master-slave 
positioning replica or a rate joystick. 

In the lower figure a computer is added to the teleoperator, and for 
short periods and limited circumstances this teleoperator may function 
autonomously. 

At the bottom are generic definitions of teleoperator and supervisory 
control. The upper drawing portrays the former without the latter. The 
lower drawing is the combination. 

In supervisory control the teleoperator's (remote) computer comwni* 
cates at high bit-rate with the teleoperator** sensors and actuators. But 
because of bandwidth constraints on the signal transmission link, or be- 
cause of teleoperator sensing limitations, communication may be restricted 
to low-bit-rate with the human operator's (local) computer. Bor this reason, 
and also because of the intermittent nature of human monitoring and repro- 
graming of commands on a keyboard (and possibly Joystick or other controls), 
the human supervisor's communication with the teleoperator tends to be at a 
slow rate, i.e., intermittent symbol strings or movement sequences on a 
master-controller with relatively many bits per instruction package. His 
communication with the local computer to refresh TV images or to edit or 
"dry run" his commands on a model before committing them to action may be 
constrained only by his own speed limitations. 

The physical separation of local and remote computer is Tot necessary 
in aircraft, industrial plants or other systems where the operator is 
physically nearby, and where supervisory control is used for reasons other 
than physical remoteness and limited communication channel capacity be- 
tween human operator and the object of control. In such situations super- 
visory control may be advantageous, nevertheless, to achieve faster or more 
accurate control, or to control simultaneously in more degrees-of-freedam 
than the operator can achieve by direct servo-control, or to relieve him 
of tedium. The latter reasons for supervisory control can apply to wor- 
ses vehicles when the human operator is not physically distant (as with 
manned submersible*) or to undersea teleoperators when a reliable high- 
bandwidth communication channel (wire or optical tether) is available. 


2. Whv Teleooorators Underseas? T*TT W&& Si BiSSO. aut ** r * m **- 

The principal reasons for interest in using teleoperators for underseas 
tasks are dollar costs and safety. 

Operations, including exploration, inspection, construction, maintenance, 
salvage and rescue, are having to be performed at increasing depths. At 
such depths - below, say 300m. (depending upon the paticular task) the time 
required for divers - mostly compression/decompression time - becomes ex- 
cessive; factors having to do with depth per ae, Including life support e<piip- 
ment, become increasingly costly; personal safety is more and more difficult 
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to maintain. These assertions are borne out by rather alarming mortality 
figures for commercial divers in the North Sea. 

Water turbidity and other depth-related factors may require greater 
bottom-time, thus compounding the decompression-time factor. Under such 
conditions, a fixed-capability teleoperator * which sometimes is seen as too 
clumsy by comparison to a human diver at shallower depths, becomes much 
more attractive economically. 

Happily, there is progress in the development of teleoperators, and 
they are becoming less clumsy. Inspection and manipulation tasks Which 
simply could not be accomplished a few years ago are now achievable, due 
to steady progress in the design of video systems, mechanical valves and 
actuators, etc. For the immediate future, however, the primary technolog- 
ical factor which is changing the prospects for undersea teleoperation is 
the computer. 

Circa 1970 divers seemed to have the edge on manned work-vehicles with 
manipulators in terms of maneuverability, manipulation, tactile sensing, 
and covertness. Because of smaller unmanned vehicles and eventually through 
unmanned untetl.ered vehicles, however, the diver (especially the tethered 
diver) is losing his edge. Manipulation, sensing and cognition remain the 
primary advantages for the diver, but the computer is changing these also. 

The comparison between teleoperators and manned submersibles is more 
clear-cut. The fact is that television cameras can now "see" with less 
light than the human eye, and new sonic imaging systems can see through 
densely turbid waters where neither human vision nor video can function. 
Spatial resolution or video can be made to approximate that of the eye by 
focusing. Present advantages of manned submersibles or teleoperators as 
work vehicles (neglecting for the moment personnel rescue) are: steropsis 
for close-up objects, and the ability of a human observer with a wide angle 
of view to keep track of the relative location of different objects. As 
the communication channel improves, to the point where the manipulator 
itself is the limiting factor, a man in a submersible can control manipu- 
lators or video pan-tilt controls just as well as a man on the surface. 

The major difference remaining between manned submersible and teleoperator 
are then cost and safety, as with the diver. The pressure vessel and life- 
support equipment make the manned submersible much more costly than the 
same vehicle without the pressure vessel and life-support equipment but 
with remote control instead. The factors of quality and reliability of 
communication and remote control then become the key factors. 


3. Why Supervisory Control of Teleoperators Underseas? Some Assertions 
about the Problem. 


a. Demands are increasingly stringent in terms of depth, sensory resolu- 
tion, speed and accuracy of power of response for accomplishment of under- 
sea tasks. Some of these tasks are always the same and are amenable to 
fixed automation, but many are different each time they occur and therefore 
cannot be done by fixed automation. 
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b. in term of depth and skill human divers are reaching their limits, 
or when they go beyond these limits they do so at significant risk to life 
and cost in support equipment and personnel. 

Teleoperators, i.e.,submersibles having video and other sensors, actu- 
ators for mobility and manipulation, and remotely controlled by human oper- 
ators, offer much promise for extending man’s flexible, adaptable, perceiv- 
ing and control capabilities into remote and hazardous environments. 

d* Present teleoperators are quite limited in sensory capability (e.g.,in 
turbid water), in manipulation capability (in speed and dexterity as compared 
to human hands), and in dealing with distortion in man-machine communication 
(misorientation of teleoperator to human body, time delays and noise). 

e. Computers are rapidly getting smaller in size and power requirement 
and cheaper in cost for a given computing capability. 

f. While accomplishment of one-of-a-kind undersea tasks by intelligent 
and cospletely autonomous robots may have appeal, we simply do not have 
available at this time such devices or the understanding to build such de- 
vices. 

g. Undersea systems, like aerospace systems, demand conservative design 
because unreliability poses severe costs. 

h. The most immediate and reliable approach would appear to be to add 
modest computer aiding and "artificial intelligence" to teleoperators, re- 
taining human sensing, motor, memory and decision capability, at least for 
higher level planning, decision-making, and control. 

i. Over a longer period of years, as computer control and artificial in- 
telligence become more sophisticated, certain human functions in teleoper- 
ation may be replaced, but greater need and demand will be placed upon 
other human functions, and in these respects the need for improved man- 
ccmputer interaction will increase, not diminish. 


Relative Roles of Man an d Computer, and Man-Computor Communication . 

In analyzing the relationship between human operator and computor in 
teleoperation, it is useful to consider how human behavi oral components, 
through two basic forms of communication are used in four human supervisory 
roles. Figure 2 summarizes the situation by arrows indicating causality”* 
between descriptors. 

Commanding the computor is done by either typing strings of symbols 
or pushing dedicated buttons or switches ( symbolic commands) or moving a 
joystick or replica controller, where there is a geometric isomorphism be- 
tween control movement and its meaning ( analogic c ommands). Observina can 
also be of symbolic displays (alphanumsrics) or analogic displays (pictures 
or geometric diagrams). Imagining ( interna 1 mental visualization) may also 
be symbolic or analogic. 
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Figure 2. Relative Roles of Man and Computer 
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Supervisory commands may be for purposes of planning information, 
such as referencing computer memory or testing a potential future action 
on a model; they may be for ordering teleoperator action ; they may be 
for making computer adjustments while monitoring tel eoperator action; or 
they may be for intervening in teleoperator action to assume direct manual 
control* Observation of displays is indicated in both the planning and 
the monitoring role of the operator* 

As shown in the fourth row, the teleoperator may be ordered to act 
in two different ways* One is Where the man shares control with the com- 
puter, i*e* the two work on the same task at the same time* The other is 
Where the man trades_control on all or some of the tasks, i*e* for those 
tasks he gives over complete control to the computer* Intervention means 
that the computer trades control back to the man* When control has been 
traded to the computer, part of the act of monitoring is to observe (analogic 
and symbolic) displays of its performance, as shown by the upward arrows* 

The computer's alternative operating roles when sharing control with 
the human operator are relieve * i*e« do things which make his work easier, 
and extend* i.e. pushing his performance beyond where it would normally be* 
When control is traded to it the computer may backup the operator by being 
ready to take over in case he fails, or it may replace him altogether* 

While this taxonomy of relationships at the present has no correspond- 
ing quantitative theory, it has been useful to the authors in. thinking 
about what is desirable for man-computer control of teleoperators* In par- 
ticular it has helped us think through the various forms of computer aiding 
which might be programmed into an experimental system. And it has clarified 
for us the potential of using a combination of general purpose typewriter 
and dedicated on special-purpose keyboard commands (symbolic) and force- 
reflecting master-slave and rate comnands (analogic)* 


5* SUPERMAN: a System for Supervisory Manipulation 

A brief description of a thesis by T.L. Brooks in progress at the Man- 
Machine Systems Lab at MIT is given on the following pages as an example 
of a supervisory manipulator system* This system is called SUPERMAN* Fig- 
ure 3 shows the general relationships between the multiple inputs (keyboard, 
dedicated symbolic keys, and analog inputs), the computer states (STANDBY, 
DEFINE, EDIT, EXECUTE, AND TAKEOVER) and the control modes (RATE, MIXED 
MASTER/SLAVE AND RATE, MASTER/SLAVE, and COMPUTER control). 

STANDBY State - When the computer is in this state, control resides 
with the main program and the operator* By pressing the proper button on 
the control console, the user can enter a particular manual control mode 
or another computer state (see Figure 4)* 

Manual Control Mode - A manual control mode is the method through 
Which the user analogically interacts with the arms* A control mode is 
independent of the state. For example, the control mode might be MASTER/ 
SLAVE while the state is EDIT* There are three kinds of modes: 
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a * RATE “ The individual degrees of freedom are controlled 
through rate commands by switches on the control console and a poten- 
^ or ra te adjustment* Both rate and resolved-motion rate are 


b) MIXES MASTER/SLAVE AND RATE - The master acts as a spring- 
loaded^ joystick in the X, Y and 2 axes, giving rate commands to the X, 

Y and z axes of the slave proportional to displacement of the master. 

I The rate of the slave arm is then reflected in the force feedback 
level which the operator feels in the master.) Both rate and resolved 
motion rate control are available. The remaining degrees of freedom, 
the left and right elevation, the azimuth and the end-effector are con- 
trolled in a master/slave mode. 


c) MASTER/SLAVE - The slave arm is driven to duplicate in position 
the action of the master. Any force felt by the slave is reflected to 
the master giving the operator force feedback (i.e. proportional to 
position disparity between master and slave). 


P E PINE - DEFINE is the primary state through which the operator enters 
a string of commands to be executed. Commands are entered by pressinq 
fP ec ^ ly dedicated buttons for each function. All of the buttons used in 
1*1 ^5? State have dual functions < 8ee Figure 4 - dual function buttons 


\ AS the title ***"• the string of contends is exe- 
cuted through this state. During the execution of the command register. 

^ ° P ! rat ° r d ! 8 , ire f to *** control, there are two methods available. 
The operator can take immediate control: (1) by pulling on the appropriate 
^* e * the MASTER in the case of MASTER/SIAVE or MIXED 
MASTER/SLAVE AND RATE modes or the rate switches in the RATE mode), or 
(2) by pressing the STOP button (all action ceases after the STOP button 
* c S?fm»Sf* SSe ? Until the operator signals for continuation or return 
I fl K? or J !an execut e a string of commands which have 
* 1*1* f fJ e hy pressing one of the lighted TASK FILE buttons, 
e operator also has the option of executing the current command reqister 
by pressing the EXECUTE button. This allows the operator to define a 
Jrf*? ° f C0CTnands «d immediately execute them to determine if any modi- 

After thC operator 18 8ure the command string 
performs the <lesire d function correctly, that function can then be saved 
as a task file or a named file. 

T AKEOVER State - TAKEOVER is a transition state between control modes, 
i.e. from computer control to the control mode in effect before the EXECUTE 
command. Special problems result during this state due to the mismatch 

* •£*£**• , j£ v * ** th f «“<**• tokeovr. The diamond 
in Figure 3 signifies that after the mismatch has been dissolved, the op- 

EXECUTiw S sSte! P ^ ° f m ° Ving ^ th ® STAN0BY 8tate o** continuing the 

The detailed meanings of the DEFINE buttons 0-15 are given below: 


BUTTON 

Number Command 

0 END 

Final command used to signal completion of DEFINE state. 

1 SAVE 

Used to save the command register on the disk as either 
a task file or a named file. A task file can be recalled 
o.nly by one of eight buttons In the STANDBY state, whereas 
a named file Is saved under a user-designated title and 
can only be recalled by the same name through the GET but- 
ton (6) In the DEFINE state. 

2 EDIT 

The EDIT command allows the user to modify the command 
register. The following options are available through the 
keyboard after entering the EDIT state; 

a) CHANGE A LINE 

b) INSERT A LINE 

c) DUPLICATE A LINE 

d) DELETE A LINE 

e LIST COMMAND REGISTER 
f) RETURN TO DEFINE 


3 2ND 

Used to enter the second function of dual coitnand keys. 

The first function of each key Is printed In black letters 
above the button. The second function is written below 
the button In gold letters. To enter a second function 
command, press the 2ND key and then the desired second com- 
mand. 

4 ERASE LAST LINE [ERASE] 

Used to erase the last entry In the command register, 

5 GET 

Used to retrieve a named command file from the disk. 

GET asks for the name of the command file to be recalled 
and then locates the file, reads it into the command register 
(and returns to DEFINE state). 

6 RESET 

Used to initialize the necessary internal variables and the 
command register to zero. 
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7 THROUGH PATH [TPATH] 

Records the present position of the arm for use in EXECUTE 
as a through point. (A through point is a position which 
the operator desires the arm to move through without stop- 
ping, i.e. non-zero velocity point.) 

8 INCREMENT (DOF) (XXXX) 

Makes an incremental motion in the desired degree of 
freedom by a selected value. The user enters. the INCREMENT 
command, then the degree of freedom (DOF), adjusts the 
desired increment (XXXX) through the potentiometer and 
presses the READ POT VALUE button directly beneath the 
potentiometer, 

9 IF (DOF) FORCE. GT, 

EXECUTE NEXT COMMAND 

If the force level in the desired degree of freedom 
(DOF) is greater than the level set by the operator (XXXX) 
the following command is executed. If the force level is 
less than the level set by the operator, the command immedi- 
ately following the IF FORCE, GT. statement is skipped during 
execution. The user enters the IF FORCE. GT. command, then 
t»,3 desired degree of freedom, adjusts the force level through 
the potentiometer. 

10 GRASP WITH FORCE (XXXX) [GRASP] 

The user enters the GRASP command and adjusts the force 
level through the potentiometer. 

11 DISCRETE PATH [DPATH] 

Records the present position of the arm for use in 
EXECUTE as a stopping point. During execution, the 
slave arm is moved from its current position to 
the recorded position with zero final velocity. 

12 LABEL (XXXX) 

Labels a position in the command register which can be 
returned to through a GOTO command. The user presses the 
LABEL button and then the number (XXXX) of the desired 
label. 

13 GOTO (XXXX) 

GOTO is a conditional command which moves to label (XXXX), 

unless the operator signals during execution to change the 
branch to (YYYY) by pressing a different button. To enter 
the command the operator presses the GOTO button and then 
the number (XXXX) of the label to which GOTO should branch. 


354 


14 

15 


2ND - 0 


2ND - 1 


2ND - 2 
through 
2ND - 15 


OPEN 

Open jaws. 


tUNiiNUUUS PATH [CPATHJ 

Records the position of the master manipulator everv 0 1 

inEXECUTE ' A continuous path Is achieved 
by interpolating between the recorded positions. 

ABSOLUTE 

Informs the execution compiler that the command register 
is to be executed exactly as recorded (see RELATIVE) The 

k S +r abso ^ ute command by pressing the 2ND 

button (#4) and then the ABSOLUTE button (#0). 

RELATIVE 

Informs the execution compiler that the positions In the 
command register are to keep the same relative displacement 
X2 r h esp «f * each other, but are to be twIsfoSid IT 
that the first position following the RELATIVE conmand 

execution^ aVaS??!’*' 0 " °t tbe 2. 

ABSOLUT? ;nJl*^ LAT J!u 5? nimand can b e cancelled by an 
absolute command, with the result that onlv the n mim am 

between the RELATIVE end ABSOLUTE winends are tfSlfeSL 

bu?ton e (»n e ^ eS . the 2® butt0 " < M > »"<! then CrelahVE - 

button (#1) to enter the command in the register. 


not assigned. 


o» . of tt” 1 * to ^ * »« •« 

major sections, one removes the nut a!d be broken down into two 

Since the user would p£f« on! Tt 0ther places U in the box. 

regardless of the orientation of the »nATTin ? t0 be U8ed for a11 nuts 

be the first comnand in the reals ter (thL dpt^attup^ cownand “bould obvious] 
following commands ar^n^ a11 «» 

register for the nut removal croon.™ ? • 71,6 entire comnand 

•—* *«— ““i zssJsrxzgL'is x°n; *• 


[BUTTON PUSH) 

(POT READINGS) 
"KEYBOARD COMMANDS" 
COMPUTER REPLIES. 
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1 [RELATIVE] 

2 [LABEL] [1] 

3 [DPATH] 


4 [GRASP] (200) 

5 [DPATH] 

6 [INCREMENT] [Y](300) 

7. [IF FORCE. GT. ] [Y] (100) 

8 [GOTO] [2] 

9 [GOTOJ [3] 

10 [LABEL] [2] 

11 [INCREMENT] [Y] (-300) 

12 [OPEN] 

13 [GOTO] [1] 


Place the slave on a nut and record 
that position by pressing the DPATH 
button. 


Turn the end effector 180* and record 
the position. 

Increment the slave by 300 counts in the 
direction that would pull the nut off. 

If the force Is greater than 100 in the 
Y direction, the nut is still on the 
bolt, therefore execute the next 
command . 


If the force had been less than 100 in 
the Y direction, the nut is free and 
this command would be executed. 


Return the arm to position before 
incrementing in if 6 . 

Release the nut- 

Return to LABEL 1 and continue turning 
the nut . 


14 [LABEL] [3] End of the first part of task - nut 

is off. 

[SAVE] "NUT-OFF" Save command register as the named 

file "NUT-OFF" (typed in at the keyboard). 


The second part of the task requires the manipulator to place the nut 
in a box. The entire command register for the program to put the nut in the 
box would be as follows: 
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1 ' [absolute] 

2 [TPATH] 

3 [DPATHJ 

* [OPEN] 

5 [TPATH] 

[SAVE] "NUT-IN-BOX” 


place°* W ° Uld alwaya be ln the sane 

bhe 8lave to a position just over 
and above the outside edge of the bo!! 

the Str 100 by Pre88lng 

center of"tbTb«! and°?ecord the* th * 
position. 


Enter same position 
duplicating line 2. 


a* in #2 by 


Th. NOMr^program T*'" “• 

operator .. soon a. the nut ua.^re. a!. 2 “ *" d r * tunl to th. 

(*•«♦» a named file) requires that th. But the praaent »tatus of each file 
obtain the file to exS £ if tu oparator type in each name to 
conoands the file will be saved as perforna the following 

the touch of a button: e immediately executed 

[GET] "NUT-OFF" 


"NUT-IN-BOX" 

" 1U " ply by th. too file, together ., on. m,. 

[SAVE] "TASK-FILE" 

and press the button which will retriev. th. *-»i / 

remove a nut and put it in the box^h! th flle (e *8*» button #1), To 
button, the execuiion conjile? tlanlflrl'li^l 8l " ply praaaaa the sane 
relative to the position of the slave at th l^if*!** half ° f the ra 8*«ter 
and then executes the program. After th* *!!»»• 1 ? 8t8nt the button *■ Pressed 

box the slave returns to the operator’s oositiin r *!i 0V u d * nd placed ln the 
control. operator a position and the computer relinquiehe 
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